Apparatus and method for capturing and blending multiple images for high-quality flash photography using mobile electronic device

ABSTRACT

A method includes capturing multiple ambient images of a scene using at least one camera of an electronic device and without using a flash of the electronic device. The method also includes capturing multiple flash images of the scene using the at least one camera of the electronic device and during firing of a pilot flash sequence using the flash. The method further includes analyzing multiple pairs of images to estimate exposure differences obtained using the flash, where each pair of images includes one of the ambient images and one of the flash images that are both captured using a common camera exposure and where different pairs of images are captured using different camera exposures. In addition, the method includes determining a flash strength for the scene based on the estimate of the exposure differences and firing the flash based on the determined flash strength.

TECHNICAL FIELD

This disclosure relates generally to image capturing systems. Morespecifically, this disclosure relates to an apparatus and method forcapturing and blending multiple images for high-quality flashphotography using a mobile electronic device.

BACKGROUND

Many mobile electronic devices, such as smartphones and tabletcomputers, include cameras that can be used to capture still and videoimages. While convenient, cameras on mobile electronic devices typicallysuffer from a number of shortcomings, including poor performance inlow-light situations. For example, some mobile electronic devices simplyuse a flash when capturing low-light images. However, the flashes usedin mobile electronic devices typically act as point sources of brightlight (not diffuse sources of light), so the use of a flash typicallycauses over-exposure or “blow out” for nearby people or objects andunder-exposure of the background. In other words, the use of a flashcreates non-uniform radiance in the images, resulting in low aestheticquality. The captured images also tend to have a bluish cast, which isnot constant across the images and therefore not easily removable. Othermobile electronic devices attempt to combine multiple images together toproduce more aesthetically-pleasing images. However, these approachesoften suffer from unnatural saturation artifacts, ghosting artifacts,color twisting, bluish color casts, or noise.

SUMMARY

This disclosure provides an apparatus and method for capturing andblending multiple images for high-quality flash photography using amobile electronic device.

In a first embodiment, a method includes capturing multiple ambientimages of a scene using at least one camera of an electronic device andwithout using a flash of the electronic device. The method also includescapturing multiple flash images of the scene using the at least onecamera of the electronic device and during firing of a pilot flashsequence using the flash. The method further includes analyzing multiplepairs of images to estimate exposure differences obtained using theflash, where each pair of images includes one of the ambient images andone of the flash images that are both captured using a common cameraexposure and where different pairs of images are captured usingdifferent camera exposures. In addition, the method includes determininga flash strength for the scene based on the estimate of the exposuredifferences and firing the flash based on the determined flash strength.

In a second embodiment, an electronic device includes at least onecamera, a flash, and at least one processing device. The at least oneprocessing device is configured to capture multiple ambient images of ascene using the at least one camera and without using the flash. The atleast one processing device is also configured to capture multiple flashimages of the scene using the at least one camera and during firing of apilot flash sequence using the flash. The at least one processing deviceis further configured to analyze multiple pairs of images to estimateexposure differences obtained using the flash, where each pair of imagesincludes one of the ambient images and one of the flash images that areboth captured using a common camera exposure and where different pairsof images are captured using different camera exposures. In addition,the at least one processing device is configured to determine a flashstrength for the scene based on the estimate of the exposure differencesand fire the flash based on the determined flash strength.

In a third embodiment, a non-transitory machine-readable medium containsinstructions that when executed cause at least one processor of anelectronic device to capture multiple ambient images of a scene using atleast one camera of the electronic device and without using a flash ofthe electronic device. The medium also contains instructions that whenexecuted cause the at least one processor of the electronic device tocapture multiple flash images of the scene using the at least one cameraof the electronic device and during firing of a pilot flash sequenceusing the flash. The medium further contains instructions that whenexecuted cause the at least one processor of the electronic device toanalyze multiple pairs of images to estimate exposure differencesobtained using the flash, where each pair of images includes one of theambient images and one of the flash images that are both captured usinga common camera exposure and where different pairs of images arecaptured using different camera exposures. In addition, the mediumcontains instructions that when executed cause the at least oneprocessor of the electronic device to determine a flash strength for thescene based on the estimate of the exposure differences and fire theflash based on the determined flash strength.

Other technical features may be readily apparent to one skilled in theart from the following figures, descriptions, and claims.

Before undertaking the DETAILED DESCRIPTION below, it may beadvantageous to set forth definitions of certain words and phrases usedthroughout this patent document. The terms “transmit,” “receive,” and“communicate,” as well as derivatives thereof, encompass both direct andindirect communication. The terms “include” and “comprise,” as well asderivatives thereof, mean inclusion without limitation. The term “or” isinclusive, meaning and/or. The phrase “associated with,” as well asderivatives thereof, means to include, be included within, interconnectwith, contain, be contained within, connect to or with, couple to orwith, be communicable with, cooperate with, interleave, juxtapose, beproximate to, be bound to or with, have, have a property of, have arelationship to or with, or the like.

Moreover, various functions described below can be implemented orsupported by one or more computer programs, each of which is formed fromcomputer readable program code and embodied in a computer readablemedium. The terms “application” and “program” refer to one or morecomputer programs, software components, sets of instructions,procedures, functions, objects, classes, instances, related data, or aportion thereof adapted for implementation in a suitable computerreadable program code. The phrase “computer readable program code”includes any type of computer code, including source code, object code,and executable code. The phrase “computer readable medium” includes anytype of medium capable of being accessed by a computer, such as readonly memory (ROM), random access memory (RAM), a hard disk drive, acompact disc (CD), a digital video disc (DVD), or any other type ofmemory. A “non-transitory” computer readable medium excludes wired,wireless, optical, or other communication links that transporttransitory electrical or other signals. A non-transitory computerreadable medium includes media where data can be permanently stored andmedia where data can be stored and later overwritten, such as arewritable optical disc or an erasable memory device.

As used here, terms and phrases such as “have,” “may have,” “include,”or “may include” a feature (like a number, function, operation, orcomponent such as a part) indicate the existence of the feature and donot exclude the existence of other features. Also, as used here, thephrases “A or B,” “at least one of A and/or B,” or “one or more of Aand/or B” may include all possible combinations of A and B. For example,“A or B,” “at least one of A and B,” and “at least one of A or B” mayindicate all of (1) including at least one A, (2) including at least oneB, or (3) including at least one A and at least one B. Further, as usedhere, the terms “first” and “second” may modify various componentsregardless of importance and do not limit the components. These termsare only used to distinguish one component from another. For example, afirst user device and a second user device may indicate different userdevices from each other, regardless of the order or importance of thedevices. A first component may be denoted a second component and viceversa without departing from the scope of this disclosure.

It will be understood that, when an element (such as a first element) isreferred to as being (operatively or communicatively) “coupled with/to”or “connected with/to” another element (such as a second element), itcan be coupled or connected with/to the other element directly or via athird element. In contrast, it will be understood that, when an element(such as a first element) is referred to as being “directly coupledwith/to” or “directly connected with/to” another element (such as asecond element), no other element (such as a third element) intervenesbetween the element and the other element.

As used here, the phrase “configured (or set) to” may be interchangeablyused with the phrases “suitable for,” “having the capacity to,”“designed to,” “adapted to,” “made to,” or “capable of” depending on thecircumstances. The phrase “configured (or set) to” does not essentiallymean “specifically designed in hardware to.” Rather, the phrase“configured to” may mean that a device can perform an operation togetherwith another device or parts. For example, the phrase “processorconfigured (or set) to perform A, B, and C” may mean a generic-purposeprocessor (such as a CPU or application processor) that may perform theoperations by executing one or more software programs stored in a memorydevice or a dedicated processor (such as an embedded processor) forperforming the operations.

The terms and phrases as used here are provided merely to describe someembodiments thereof, but not to limit the scope of other embodiments ofthis disclosure. It is to be understood that the singular forms “a,”“an,” and “the” include plural references unless the context clearlydictates otherwise. All terms and phrases, including technical andscientific terms and phrases, used here have the same meanings ascommonly understood by one of ordinary skill in the art to which theembodiments of this disclosure belong. It will be further understoodthat terms and phrases, such as those defined in commonly useddictionaries, should be interpreted as having a meaning that isconsistent with their meaning in the context of the relevant art andwill not be interpreted in an idealized or overly formal sense unlessexpressly so defined here. In some cases, the terms and phrases definedhere may be interpreted to exclude embodiments of this disclosure.

Examples of an “electronic device” according to embodiments of thisdisclosure may include at least one of a smartphone, a tablet personalcomputer (PC), a mobile phone, a video phone, an e-book reader, adesktop PC, a laptop computer, a netbook computer, a workstation, apersonal digital assistant (PDA), a portable multimedia player (PMP), anMP3 player, a mobile medical device, a camera, or a wearable device(such as smart glasses, a head-mounted device (HMD), electronic clothes,an electronic bracelet, an electronic necklace, an electronicappcessory, an electronic tattoo, a smart mirror, or a smart watch).Definitions for other certain words and phrases may be providedthroughout this patent document. Those of ordinary skill in the artshould understand that in many if not most instances, such definitionsapply to prior as well as future uses of such defined words and phrases.

None of the description in this application should be read as implyingthat any particular element, step, or function is an essential elementthat must be included in the claim scope. The scope of patented subjectmatter is defined only by the claims. Moreover, none of the claims isintended to invoke 35 U.S.C. § 112(f) unless the exact words “means for”are followed by a participle. Use of any other term, including withoutlimitation “mechanism,” “module,” “device,” “unit,” “component,”“element,” “member,” “apparatus,” “machine,” “system,” “processor,” or“controller,” within a claim is understood by the Applicant to refer tostructures known to those skilled in the relevant art and is notintended to invoke 35 U.S.C. § 112(f).

BRIEF DESCRIPTION OF THE DRAWINGS

For a more complete understanding of this disclosure and its advantages,reference is now made to the following description taken in conjunctionwith the accompanying drawings, in which like reference numeralsrepresent like parts:

FIG. 1 illustrates an example network configuration including anelectronic device in accordance with this disclosure;

FIG. 2 illustrates an example process for multi-pair image analysis forflash control in a mobile electronic device in accordance with thisdisclosure;

FIGS. 3 and 4 illustrate example analysis operations for analyzingexposure differences in the process of FIG. 2 in accordance with thisdisclosure;

FIG. 5 illustrates an example process for multi-scale blending of imagesin a mobile electronic device in accordance with this disclosure;

FIG. 6 illustrates an example process for an image registrationoperation in the process of FIG. 5 in accordance with this disclosure;

FIGS. 7 and 8 illustrate an example process for an exposure analysisoperation in the process of FIG. 5 in accordance with this disclosure;

FIGS. 9, 10, 11, 12, and 13 illustrate an example process for an imagede-ghosting operation in the process of FIG. 5 in accordance with thisdisclosure;

FIGS. 14 and 15 illustrate an example process for an image blendingoperation in the process of FIG. 5 in accordance with this disclosure;

FIG. 16 illustrates an example process for a contrast enhancementoperation in the process of FIG. 5 in accordance with this disclosure;

FIG. 17 illustrates an example method for multi-pair image analysis andmulti-scale blending in accordance with this disclosure; and

FIGS. 18, 19, 20, and 21 illustrate example results that can be obtainedusing multi-pair image analysis and multi-scale blending in accordancewith this disclosure.

DETAILED DESCRIPTION

FIGS. 1 through 21, discussed below, and the various embodiments of thisdisclosure are described with reference to the accompanying drawings.However, it should be appreciated that this disclosure is not limited tothese embodiments, and all changes and/or equivalents or replacementsthereto also belong to the scope of this disclosure. The same or similarreference denotations may be used to refer to the same or similarelements throughout the specification and the drawings.

As noted above, many mobile electronic devices suffer from a number ofshortcomings, including poor performance in low-light situations. Somemobile electronic devices simply use a flash for capturing low-lightimages, which typically results in non-uniform radiance and lowaesthetic quality. Other mobile electronic devices attempt to combinemultiple images together to produce more aesthetically-pleasing imagesbut suffer from problems such as unnatural saturation artifacts,ghosting artifacts, color twisting, bluish color casts, or noise.

This disclosure provides techniques for using multiple images capturedusing a flash by combining principles of multi-frame high dynamic range(HDR) imaging, where camera exposure settings are adjusted to capturemultiple images in the presence of the flash. This is accomplished byanalyzing pairs of images captured by an electronic device to determinehow to control a flash of the electronic device. Multiple images arethen captured by the electronic device based on the flash control, andthose images are processed and blended to produce a final image having amore uniform radiance. This may allow, for example, moreaesthetically-pleasing images having more natural colors in low-lightsituations to be produced. These images may suffer from little or noblow-out and may have backgrounds and foregrounds that are more evenlyilluminated. These images may also suffer from less saturationartifacts, ghosting artifacts, color twisting, bluish color casts, ornoise.

FIG. 1 illustrates an example network configuration 100 including anelectronic device in accordance with this disclosure. The embodiment ofthe network configuration 100 shown in FIG. 1 is for illustration only.Other embodiments of the network configuration 100 could be used withoutdeparting from the scope of this disclosure.

According to embodiments of this disclosure, an electronic device 101 isincluded in the network environment 100. The electronic device 101 caninclude at least one of a bus 110, a processor 120, a memory 130, aninput/output (I/O) interface 150, a display 160, a communicationinterface 170, or a sensor 180. In some embodiments, the electronicdevice 101 may exclude at least one of these components or may add atleast one other component. The bus 110 includes a circuit for connectingthe components 120-180 with one another and for transferringcommunications (such as control messages and/or data) between thecomponents.

The processor 120 includes one or more of a central processing unit(CPU), an application processor (AP), or a communication processor (CP).The processor 120 is able to perform control on at least one of theother components of the electronic device 101 and/or perform anoperation or data processing relating to communication. In someembodiments, the processor 120 can be a graphics processor unit (GPU).For example, the processor 120 can receive image data captured by atleast one camera during a capture event. The processor 120 can processthe image data (as discussed in more detail below) to perform multi-pairimage analysis and multi-scale blending.

The memory 130 can include a volatile and/or non-volatile memory. Forexample, the memory 130 can store commands or data related to at leastone other component of the electronic device 101. According toembodiments of this disclosure, the memory 130 can store software and/ora program 140. The program 140 includes, for example, a kernel 141,middleware 143, an application programming interface (API) 145, and/oran application program (or “application”) 147. At least a portion of thekernel 141, middleware 143, or API 145 may be denoted an operatingsystem (OS).

The kernel 141 can control or manage system resources (such as the bus110, processor 120, or memory 130) used to perform operations orfunctions implemented in other programs (such as the middleware 143, API145, or application program 147). The kernel 141 provides an interfacethat allows the middleware 143, the API 145, or the application 147 toaccess the individual components of the electronic device 101 to controlor manage the system resources. The application 147 includes one or moreapplications for image capture as discussed below. These functions canbe performed by a single application or by multiple applications thateach carries out one or more of these functions. The middleware 143 canfunction as a relay to allow the API 145 or the application 147 tocommunicate data with the kernel 141, for instance. A plurality ofapplications 147 can be provided. The middleware 143 is able to controlwork requests received from the applications 147, such as by allocatingthe priority of using the system resources of the electronic device 101(like the bus 110, the processor 120, or the memory 130) to at least oneof the plurality of applications 147. The API 145 is an interfaceallowing the application 147 to control functions provided from thekernel 141 or the middleware 143. For example, the API 145 includes atleast one interface or function (such as a command) for filing control,window control, image processing, or text control.

The I/O interface 150 serves as an interface that can, for example,transfer commands or data input from a user or other external devices toother component(s) of the electronic device 101. The I/O interface 150can also output commands or data received from other component(s) of theelectronic device 101 to the user or the other external device.

The display 160 includes, for example, a liquid crystal display (LCD), alight emitting diode (LED) display, an organic light emitting diode(OLED) display, a quantum-dot light emitting diode (QLED) display, amicroelectromechanical systems (MEMS) display, or an electronic paperdisplay. The display 160 can also be a depth-aware display, such as amulti-focal display. The display 160 is able to display, for example,various contents (such as text, images, videos, icons, or symbols) tothe user. The display 160 can include a touchscreen and may receive, forexample, a touch, gesture, proximity, or hovering input using anelectronic pen or a body portion of the user.

The communication interface 170, for example, is able to set upcommunication between the electronic device 101 and an externalelectronic device (such as a first electronic device 102, a secondelectronic device 104, or a server 106). For example, the communicationinterface 170 can be connected with a network 162 or 164 throughwireless or wired communication to communicate with the externalelectronic device. The communication interface 170 can be a wired orwireless transceiver or any other component for transmitting andreceiving signals, such as images.

The electronic device 101 further includes one or more sensors 180 thatcan meter a physical quantity or detect an activation state of theelectronic device 101 and convert metered or detected information intoan electrical signal. For example, one or more sensors 180 can includeone or more buttons for touch input, one or more cameras, a gesturesensor, a gyroscope or gyro sensor, an air pressure sensor, a magneticsensor or magnetometer, an acceleration sensor or accelerometer, a gripsensor, a proximity sensor, a color sensor (such as a red green blue(RGB) sensor), a bio-physical sensor, a temperature sensor, a humiditysensor, an illumination sensor, an ultraviolet (UV) sensor, anelectromyography (EMG) sensor, an electroencephalogram (EEG) sensor, anelectrocardiogram (ECG) sensor, an infrared (IR) sensor, an ultrasoundsensor, an iris sensor, or a fingerprint sensor. The sensor(s) 180 canalso include an inertial measurement unit, which can include one or moreaccelerometers, gyroscopes, and other components. The sensor(s) 180 canfurther include a control circuit for controlling at least one of thesensors included here. Any of these sensor(s) 180 can be located withinthe electronic device 101. The one or more cameras can capture images asdiscussed below and are used in conjunction with at least one flash 190.The flash 190 represents a device configured to generate illuminationfor use in image capture by the electronic device 101, such as one ormore LEDs.

The first external electronic device 102 or the second externalelectronic device 104 can be a wearable device or an electronicdevice-mountable wearable device (such as an HMD). When the electronicdevice 101 is mounted in the electronic device 102 (such as the HMD),the electronic device 101 can communicate with the electronic device 102through the communication interface 170. The electronic device 101 canbe directly connected with the electronic device 102 to communicate withthe electronic device 102 without involving with a separate network. Theelectronic device 101 can also be an augmented reality wearable device,such as eyeglasses, that include one or more cameras.

The wireless communication is able to use at least one of, for example,long term evolution (LTE), long term evolution-advanced (LTE-A), 5thgeneration wireless system (5G), millimeter-wave or 60 GHz wirelesscommunication, Wireless USB, code division multiple access (CDMA),wideband code division multiple access (WCDMA), universal mobiletelecommunication system (UMTS), wireless broadband (WiBro), or globalsystem for mobile communication (GSM), as a cellular communicationprotocol. The wired connection can include, for example, at least one ofa universal serial bus (USB), high definition multimedia interface(HDMI), recommended standard 232 (RS-232), or plain old telephoneservice (POTS). The network 162 includes at least one communicationnetwork, such as a computer network (like a local area network (LAN) orwide area network (WAN)), Internet, or a telephone network.

The first and second external electronic devices 102 and 104 and server106 each can be a device of the same or a different type from theelectronic device 101. According to certain embodiments of thisdisclosure, the server 106 includes a group of one or more servers.Also, according to certain embodiments of this disclosure, all or someof the operations executed on the electronic device 101 can be executedon another or multiple other electronic devices (such as the electronicdevices 102 and 104 or server 106). Further, according to certainembodiments of this disclosure, when the electronic device 101 shouldperform some function or service automatically or at a request, theelectronic device 101, instead of executing the function or service onits own or additionally, can request another device (such as electronicdevices 102 and 104 or server 106) to perform at least some functionsassociated therewith. The other electronic device (such as electronicdevices 102 and 104 or server 106) is able to execute the requestedfunctions or additional functions and transfer a result of the executionto the electronic device 101. The electronic device 101 can provide arequested function or service by processing the received result as it isor additionally. To that end, a cloud computing, distributed computing,or client-server computing technique may be used, for example. WhileFIG. 1 shows that the electronic device 101 includes the communicationinterface 170 to communicate with the external electronic device 104 orserver 106 via the network 162, the electronic device 101 may beindependently operated without a separate communication functionaccording to some embodiments of this disclosure.

The server 106 can optionally support the electronic device 101 byperforming or supporting at least one of the operations (or functions)implemented on the electronic device 101. For example, the server 106can include a processing module or processor that may support theprocessor 120 implemented in the electronic device 101.

Although FIG. 1 illustrates one example of a network configuration 100including an electronic device 101, various changes may be made toFIG. 1. For example, the network configuration 100 could include anynumber of each component in any suitable arrangement. In general,computing and communication systems come in a wide variety ofconfigurations, and FIG. 1 does not limit the scope of this disclosureto any particular configuration. Also, while FIG. 1 illustrates oneoperational environment in which various features disclosed in thispatent document can be used, these features could be used in any othersuitable system.

FIG. 2 illustrates an example process 200 for multi-pair image analysisfor flash control in a mobile electronic device in accordance with thisdisclosure. For ease of explanation, the process 200 shown in FIG. 2 isdescribed as being performed using the electronic device 101 of FIG. 1.However, the process 200 shown in FIG. 2 could be used with any othersuitable electronic device and in any suitable system.

The process 200 is generally used to identify and make intelligentadjustments to the flash strength of the flash 190 and possibly otherparameters such as camera exposure and number of frames. The flashstrength and optionally the other parameters that are identified usingthe process 200 can then be used as described below to supportmulti-scale blending of images (which is described with respect to FIG.5). The flash strength is identified here using pilot flash sceneanalysis/subject detection. A pilot flash sequence generally refers toone or more flashes that occur prior to the main flash used to produce afinal image of a scene. A pilot flash sequence is often used inelectronic devices for other purposes as well, such as red-eyereduction, and may include a single flash or multiple flashes. The sceneanalysis/subject detection can be used to identify object types in ascene, a distance to a closest object, a scene type (such as indoor oroutdoor, night or day, or macro or wide-angle), or other characteristicsof the scene. The determined flash strength and optionally the otherparameters are determined using this information.

As shown in FIG. 2, a capture request 202 is received by the electronicdevice 101. The capture request 202 represents any suitable command orinput indicating a need or desire to capture an image of a scene usingthe electronic device 101. For example, the capture request 202 could beinitiated in response to a user's pressing of a “soft” button presentedon the display 160 or the user's pressing of a “hard” button. Inresponse to the capture request 202, the processor 120 performs acapture operation 204 using the camera of the electronic device 101 tocapture multiple ambient images 206 of the scene. An ambient imagegenerally refers to an image of a scene in which little or no light fromthe electronic device 101 is illuminating the scene, so the flash 190may not be used in the capture of the ambient images 206. In someinstances, during the capture operation 204, the processor 120 cancontrol the camera of the electronic device 101 so that the ambientimages 206 are captured rapidly in a burst mode. Different ambientimages 206 can be captured using different camera exposures.

In response to the capture request 202, the processor 120 also performsa capture operation 208 using the camera of the electronic device 101 tocapture multiple flash images 210 of the scene. A flash image generallyrefers to an image of a scene in which light from the electronic device101 is illuminating the scene, so the flash 190 is used in the captureof the flash images 210. In some instances, during the capture operation208, the processor 120 can control the camera of the electronic device101 so that the flash images 210 are captured rapidly in a burst mode.The flash 190 is used here to generate the pilot flash sequence, and theflash images 210 may be captured using a common flash strength. Theflash strength used here may denote a default flash strength or otherflash strength used by the camera or the electronic device 101.Different flash images 210 can be captured using different cameraexposures.

In this example, the ambient images 206 and the flash images 210 formmultiple ambient-flash image pairs. That is, the processor 120 cancontrol the camera so that multiple pairs of images are obtained, whereeach pair includes one ambient image 206 captured without using theflash 190 and one flash image 210 captured using the flash 190. Eachimage pair can be captured using a common camera exposure and camerasensitivity (ISO setting), and different image pairs can be capturedusing different camera exposures or camera sensitivities. It should benoted, however, that there is no need to capture the images in eachimage pair consecutively. The ambient images 206 and the flash images210 can be captured in any suitable order, as long as the processor 120obtains multiple ambient-flash image pairs.

The images 206 and 210 are used by the processor 120 in an analysisoperation 212 to identify the exposure differences that are obtained inthe scene using the flash 190. The analysis operation 212 occurs inorder to quantify the exposure differences that are obtained using theflash 190 and the different camera exposures/camera sensitivities. Inthis way, the analysis operation 212 can identify the exposuredifferences between ambient lighting and flash lighting in a scene,which could occur in any suitable manner (such as at the pixel level inthe images or for the foreground or one or more objects in the images).This information can then be used to identify the ideal or desired flashstrength for capturing an image of the scene. This information can alsobe used to perform other functions, such as color correction. Theanalysis operation 212 includes any suitable operations to identifyexposure differences between images. Two example implementations of theanalysis operation 212 are described below, although otherimplementations of the analysis operation 212 could also be used. Onebenefit of using multiple pairs of ambient/flash images is that theresulting analysis is more robust to over-exposed and under-exposedregions of the images, yielding a more accurate estimate of the exposuredifferences from the use of the flash 190.

The exposure differences identified by the analysis operation 212 areused by the processor 120 during a mapping operation 214 to map theexposure differences to a suitable flash strength. The mapping hereessentially translates the exposure differences into a suitable strengthfor the flash 190 to be used when capturing subsequent images of thescene. Here, the mapping can consider various aspects of the exposuredifferences, such as sizes of foreground regions/objects in the sceneand the sizes of background regions in the scene. The mapping can alsoconsider the types of objects in the scene, such as whether the sceneappears to include at least one person or one or more inanimate objects.The mapping can further be based on an estimated distance to the closestobject in the scene. In addition, the mapping can be based on whetherthe image is being captured indoors or outdoors, at night or during theday, or using a macro lens or a wide-angle lens. The specific mappingsused can vary based on a number of circumstances, such as the design ofthe camera being used in the electronic device 101.

The identified flash strength can optionally be used by the processor120 during a color cast determination operation 216. During thisoperation 216, the processor 120 attempts to estimate the regions of anysubsequent images where blue casting or other color casting may form asa result of the use of the flash 190 at the identified flash strength.This information can be useful in later processing of the subsequentimages to remove the casting from the subsequent images. Theidentification of the regions performed here can be based on theexposure differences identified by the analysis operation 212 and canidentify the likely areas where casting may occur based on the exposuredifferences.

A modulated flash firing 218 occurs using the identified flash strength.For example, when the processor 120 is ready to capture additionalimages of the scene in order to produce a final image of the scene, theprocessor 120 can trigger the flash 190. The additional images of thescene are then captured by the camera of the electronic device 101 whilethe scene is being illuminated using the flash 190, which operates atthe identified flash strength. Ideally, the use of the identified flashstrength allows the additional images to then be blended or otherwiseprocessed to provide a more uniform illumination in the final image ofthe scene.

FIGS. 3 and 4 illustrate example analysis operations 212 for analyzingexposure differences in the process 200 of FIG. 2 in accordance withthis disclosure. In particular, FIG. 3 illustrates an exampleimplementation of the analysis operation 212 in which the analysis isperformed using a prior model on a per-pixel basis, and FIG. 4illustrates an example implementation of the analysis operation 212 inwhich the analysis is performed using artificial intelligence. Ofcourse, other implementations of the analysis operation 212 are alsopossible and fall within the scope of this disclosure.

As shown in FIG. 3, two images 302 and 304 in an image pair are beinganalyzed. The images here include one ambient image 302 (representingone of the images 206) and one flash image 304 (representing one of theimages 210). The images 302 and 304 are subject to a division operation306, which divides the value of each pixel in one image 302 or 304 bythe value of the corresponding pixel in the other image 304 or 302. Thequotient values resulting from the division are subjected to alogarithmic operation 308 (a log 2 operation in this example) to convertthe quotient values into the logarithmic domain. A rectifier linear unit310 operates to prevent the values in the logarithmic domain from beingnegative, such as by selecting (for each value in the logarithmicdomain) the greater of that value or zero. The operations 306, 308, and310 here can be performed for each pair of ambient/flash images capturedby the electronic device 101 during the capture operations 204 and 208.

Because the ambient/flash images can be captured by the electronicdevice 101 using different camera exposures and/or camera sensitivities,different images may often have resulting data that is reliable in someareas and not reliable in other areas. The data resulting from theoperations 306, 308, and 310 for the different pairs of ambient/flashimages can therefore be averaged in an averaging operation 312, whichaverages the values obtained for the different camera exposures/camerasensitivities. The averaged values are passed through an edge-preservingfilter 314, which smooths out the averaged data and reduces noise whilepreserving edges within the averaged data. The edges could denote theedges of one or more people or objects in the foreground of the imagesor in the background of the images. Various types of edge-preservingfilters are known in the art. In some embodiments, the edge-preservingfilter 314 could represent a bilateral filter, which operates to replacethe intensity of each average pixel with a weighted average of intensityvalues from nearby average pixels. Note, however, that otherimplementations of the edge-preserving filter 314 could be used.

The outputs of the edge-preserving filter 314 are the exposuredifferences 316 obtained through the use of the flash 190. The exposuredifferences could be expressed in any suitable manner. In someembodiments, for example, the exposure differences can be expressed as agrayscale image, where darker pixels in the grayscale image identifyareas where the exposure differences were smaller and brighter pixels inthe grayscale image identify areas where the exposure differences werelarger. For instance, if the original ambient and flash images 206 and210 included a person in the foreground and a dark background, thegrayscale image would likely include many white pixels in the area ofthe images where the person was located, since the illumination from theflash 190 would greatly improve the brightness of the person in theflash images. In contrast, the grayscale image would likely include manydark pixels in the area of the images where the background was located,since the illumination from the flash 190 may not improve (or would onlyslightly improve) the brightness of the background in the flash images.

As shown in FIG. 4, once again, two images 402 and 404 of each imagepair are analyzed. In this example, however, the pairs of images 402 and404 are passed through a convolutional neural network (CNN) 406. Aconvolutional neural network 406 generally represents a type of deepartificial neural network, and convolutional neural networks are oftenapplied to analyzing images. In the convolutional neural network 406,layers of convolutional neurons apply a convolution operation thatemulates the response of individual neurons to visual stimuli. Eachneuron typically applies some function to its input values (often byweighting different input values differently) to generate output values.Pooling layers can be used to combine the output values of neuronclusters from one layer into input values for single neurons in anotherlayer.

The convolutional neural network 406 here can be used to process theimage pairs and generate exposure differences (such as in the form of agrayscale image). This can be accomplished by training the convolutionalneural network 406 so that the weights of the neurons have appropriatevalues. The convolutional neural network 406 can also be trained toperform other functions, such as specularity removal (the removal ofsmall bright spots where distant specular surfaces in the backgroundmight still yield a strong response to a flash) and de-ghosting (theremoval of movement from one image to another).

Although FIG. 2 illustrates one example of a process 200 for multi-pairimage analysis for flash control in a mobile electronic device and FIGS.3 and 4 illustrate examples of analysis operations 212 for analyzingexposure differences in the process 200 of FIG. 2, various changes maybe made to FIGS. 2, 3, and 4. For example, while shown as sequences ofsteps, various operations shown in FIGS. 2, 3, and 4 could overlap,occur in parallel, occur in a different order, or occur any number oftimes. As a particular example, the operations 204 and 208 in FIG. 2could be reversed or interleaved so that the ambient images 206 and theflash images 210 are captured in a different order than that shown here.Also, the specific analyses shown in FIGS. 3 and 4 are examples only,and other techniques could be used to identify exposure differencesinvolving any number of images.

FIG. 5 illustrates an example process 500 for multi-scale blending ofimages in a mobile electronic device in accordance with this disclosure.For ease of explanation, the process 500 shown in FIG. 5 is described asbeing performed using the electronic device 101 of FIG. 1. However, theprocess 500 shown in FIG. 5 could be used with any other suitableelectronic device and in any suitable system.

The process 500 is generally used to capture multiple images of a sceneusing different camera exposures at the same flash strength, namely theflash strength determined using the process 200 described above. In someembodiments, the different camera exposures can be achieved by varyingthe camera's sensor gain and exposure time. Generally, the electronicdevice 101 can capture one or more images having shorter exposures, oneor more images having longer exposures, and optionally one or moreimages having mid-range exposures between the shorter and longerexposures. The images are then aligned geometrically andphotometrically, and one of the images (often a mid- or longer-exposureimage) is selected as a reference. The images are blended to, amongother things, replace one or more blown-out regions in the referenceimage with one or more regions based on or extracted from other images(often the shorter-exposure images). Motion in the images can also beestimated in order to remove ghosting artifacts, and other processingcan occur to improve the final image of the scene.

As shown in FIG. 5, a collection 502 of images is captured using thecamera of the electronic device 101. Here, the collection 502 includesat least three images 504, 506, and 508, each of which can be capturedusing a different exposure but a common flash strength. For example, theimage 504 could be captured using the shortest exposure, the image 508could be captured using the longest exposure, and the image 506 could becaptured using an intermediate exposure between the shortest and longestexposures. Note, however, that other numbers of images (including twoimages or more than three images) could be captured and other numbers ofexposures (including two exposures or more than three exposures) couldbe used. One or multiple images could be captured at each exposure, andthere is no requirement that an equal number of images be captured perexposure.

The image collection 502 is provided to an image registration operation510, which generally operates to align the images 504, 506, and 508.Alignment may be needed if the electronic device 101 moves or rotates inbetween image captures and causes objects in the images to move orrotate slightly, which is common with handheld devices. The images 504,506, and 508 here can be aligned both geometrically and photometrically.In some embodiments, the image registration operation 510 can use globalOriented FAST and Rotated BRIEF (ORB) features as local features andglobal features from a block search to align the images. One exampleimplementation of the image registration operation 510 is describedbelow, although other implementations of the image registrationoperation 510 could also be used.

The aligned images are output and processed using an exposure analysisoperation 512 and a de-ghosting operation 514. The exposure analysisoperation 512 analyzes the aligned images to generate well-exposednessmaps for the aligned images. Each well-exposedness map generallyidentifies the area or areas of one of the aligned images that arewell-exposed (not over-exposed or under-exposed). Different metrics canbe used to define the well-exposed portions of the images based on thecamera exposures used to capture those images. For instance, differentmetrics can be defined by different functions, where the functionsconvert pixel values into well-exposedness values and where thefunctions are applied to different aligned images. One exampleimplementation of the exposure analysis operation 512 is describedbelow, although other implementations of the exposure analysis operation512 could also be used.

The de-ghosting operation 514 processes the aligned images to identifymotion occurring in the images, such as people or objects moving withinthe images. In some embodiments, the de-ghosting operation 514 divideseach of the aligned images into tiles, such as sixteen tiles arranged ina four-by-four grid. The de-ghosting operation 514 then processes thetiles to identify motion, where the motion is identified as differencesbetween the tiles. In this way, the de-ghosting operation 514 generatesmotion maps to identify areas in the images where motion is occurring.For instance, each motion map could include black pixels indicatingwhere no motion is detected and white pixels indicating where motion isdetected. The de-ghosting operation 514 can also equalize the images toaccount for the different camera exposures/camera sensitivities used tocapture the images. One example implementation of the de-ghostingoperation 514 is described below, although other implementations of thede-ghosting operation 514 could also be used.

A multi-scale image blending operation 516 receives the aligned images,the well-exposedness maps, and the motion maps and uses this informationto generate one or more blended images. Each blended image can includeor be based on portions of different images. For example, a blendedimage could be formed by selecting one of the images (such as an image506 captured with an intermediate exposure) as a reference image andreplacing blown-out or other portions of the reference image using orbased on corresponding portions from other images. As a particularexample, over-exposed portions of the image 506 can typically bereplaced with or using corresponding portions of the image 504 when theimage 504 is captured using a shorter exposure. The blending can alsoaccount for motion in the images, such as by avoiding the insertion of amoving object from one image in the wrong position in the referenceimage. In some embodiments, the blending represents a weighted blendingof synthesized images across multiple scales, where blending maps areused as the weights and are based on a composite of the well-exposednessmaps and de-ghosting maps. For instance, each of the blending maps couldrepresent a product of one of the well-exposedness maps and one of thede-ghosting maps. One example implementation of the multi-scale blendingoperation 516 is described below, although other implementations of themulti-scale blending operation 516 could also be used.

Each blended image can then be subjected to one or more post-processingoperations in order to improve the blended image. For example, theblended image can be subjected to an edge-enhanced noise filteringfunction 518, which generally operates to remove noise and improve theappearances of edges in the blended image. Various techniques for edgeenhancement and noise filtering are known in the art. In someembodiments, the filtering function 518 can represent a multi-scalede-noising process that is guided by the blending maps, well-exposednessmaps, and de-ghosting maps. The filtered blended image can be processedby a contrast enhancement operation 520, which generally operates toincrease the overall contrast of the blended image while maintainingnatural hue within the blended image. One example implementation of thecontrast enhancement operation 520 is described below, although otherimplementations of the contrast enhancement operation 520 could also beused.

The output of the process 500 is at least one final image 522 of thescene. The final image 522 generally represents a blend of the originalimages 504, 506, and 508 after processing. As noted above, for example,the final image 522 may represent the image selected as the referenceimage (such as the image 506), with one or more portions of thereference image (such as one or more blown-own regions) replaced orcombined with one or more corresponding portions of at least one otherimage (such as the shorter-exposure image 504). Ideally, the final image522 has both a foreground and a background with more uniformillumination. The illumination need not be completely uniform, but theillumination in the final image 522 is more uniform compared to theillumination in at least the reference image.

FIG. 6 illustrates an example process for an image registrationoperation 510 in the process 500 of FIG. 5 in accordance with thisdisclosure. As described above, the image registration operation 510 isused to align multiple images (such as the images 504, 506, and 508)captured by the electronic device 101. In FIG. 6, multiple images,namely a reference image and a non-reference image, are aligned byfitting a transformation matrix H to matched feature points. A pair ofmatched feature points represents a feature point in one image that ismatched to a corresponding feature in the other image. Overall, thishelps to compensate for movement of the camera/electronic device 101.

As shown in FIG. 6, a reference image I_(ref) and a non-reference imageI_(nonref) are provided to a feature detection and matching function602, which generally operates to identify the feature points in eachimage and match the feature points common to both images. In thisexample, the matched feature points are expressed as {p_(nonref),p_(ref)} values. The feature detection and matching function 602 can useany suitable technique for identifying and matching feature points, suchas ORB feature detection and matching. Various types of feature pointdetection and matching are known in the art. A first transformationmatrix estimation function 604 receives the matched feature points{p_(nonref), p_(ref)} and generates an initial estimate of thetransformation matrix. The initial estimate represents an initial guessof the transformation matrix that could be used to transform thefeatures points of the non-reference image to match the features pointsof the reference image. Various types of transformation matrixestimation techniques are known in the art, such as linear estimation.

The reference and non-reference images and the initial estimate of thetransformation matrix are provided to a block search function 606.Unlike the feature detection and matching (which matches featurepoints), the block search function 606 attempts to match blocks in thereference and non-reference images after at least one of the images hasbeen transformed using the initial estimate of the transformationmatrix. This allows the block search to be guided by the identifiedfeature points. In this example, the matched blocks are expressed as{q_(nonref), g_(ref)} values. The block search function 606 can use anysuitable technique for identifying and matching blocks.

A second transformation matrix estimation function 608 receives thematched feature points {p_(nonref), p_(ref)} and the matched blocks{q_(nonref), g_(ref)} and generates a final estimate of thetransformation matrix H. The final estimate ideally represents the bestestimate of the transformation matrix to be used to transform thefeatures points and blocks of the non-reference image to match thefeatures points and blocks of the reference image. Once thenon-reference image is transformed using the transformation matrix H,the non-reference image is generally aligned with the reference image.Again, various types of transformation matrix estimation techniques areknown in the art, such as linear estimation.

Note that the process shown in FIG. 6 can be repeated for eachnon-reference image in the image collection 502, typically using thesame image from the collection 502 as the reference image. The resultsfrom performance of the process in FIG. 6 is ideally a set of images(denoted 504′, 506′, and 508′ below) that are generally aligned with oneanother. It is possible that one of the images in the set of alignedimages still represents the corresponding original image, such as whenthe image 506′ matches the image 506 if the image 506 is used as thereference image during the process 600.

FIGS. 7 and 8 illustrate an example process for an exposure analysisoperation 512 in the process 500 of FIG. 5 in accordance with thisdisclosure. As described above, the exposure analysis operation 512 isused to identify different areas of aligned versions of images capturedby the electronic device 101 that are well-exposed and therefore includethe most useful and reliable image information. As shown in FIG. 7,different metrics 702, 704, and 706 are applied to the aligned images504′, 506′, and 508′. Different metrics 702, 704, and 706 can be usedhere since the images 504′, 506′, and 508′ are associated with differentcamera exposures. Thus, different measures can be used to determinewhether portions of the different images 504′, 506′, and 508′ arewell-exposed. In this way, each image 504′, 506′, and 508′ is convertedinto a well-exposedness map 708, 710, and 712, respectively. Eachwell-exposedness map 708, 710, and 712 could represent a grayscaleimage, where brighter colors represent well-exposed areas of theassociated image and darker colors represent over- or under-exposedareas of the associated image.

Examples of the metrics 702, 704, and 706 that could be used here areshown in FIG. 8, where lines 802, 804, and 806 respectively representthe functions applied by the metrics 702, 704, and 706. For eachfunction, the corresponding line 802, 804, or 806 identifies how actualpixel values in an image 504′, 506′, or 508′ are translated into valuescontained in the well-exposedness map 708, 710, or 712. For example, theline 802 here represents how pixel values in a short-exposure image(such as the image 504′) can be converted into corresponding values inthe well-exposedness map 708. The line 804 here represents how pixelvalues in a mid-exposure image (such as the image 506′) can be convertedinto corresponding values in the well-exposedness map 710. The line 806here represents how pixel values in a long-exposure image (such as theimage 508′) can be converted into corresponding values in thewell-exposedness map 712. Because the pixel values in different imagesof the same scene can vary in a generally known manner as the cameraexposure changes, it is possible to predict which pixel values are morelikely to represent well-exposed areas of images captured at differentexposures. Note, however, that the functions shown in FIG. 8 areexamples only, and other metrics could be used to convert an image intoa well-exposedness map.

FIGS. 9, 10, 11, 12, and 13 illustrate an example process for an imagede-ghosting operation 514 in the process 500 of FIG. 5 in accordancewith this disclosure. As described above, the image de-ghostingoperation 514 is used to identify motion in aligned versions of imagescaptured by the electronic device 101. As shown in FIG. 9, the imagede-ghosting operation 514 generally includes operations performed by areference frame block 902 and a main block 904. The reference frameblock 902 receives luminance (Y) values of a reference image and anon-reference image and generates a motion multiplier (Mot_Mult) for thetwo images. The motion multiplier controls how aggressively the mainblock 904 in the image de-ghosting operation 514 will be in terms ofrejecting pixels with high difference as motion. The main block 904receives the motion multiplier, the luminance values of the referenceand non-reference images, and chrominance values (U and V) of thereference and non-reference images, along with any desired tuningparameters (such as a noise level estimate denoted Sig_Est). The noiselevel estimate can be based on the ISO level of the camera during thecapture of the images. The main block 904 uses this information togenerate a de-ghosting map 906 for the two images. The de-ghosting map906 (also referred to as a motion map) identifies areas in the twoimages where motion is occurring and should be removed, therebyidentifying the expected motion and noise level in the images.

FIG. 10 illustrates an example implementation of the reference frameblock 902 in FIG. 9. As shown in FIG. 10, the reference frame block 902includes downscaling functions 1002 and 1004. The downscaling function1002 receives the luminance values Y_(ref) of the reference image anddownscales the luminance values to produce downscaled luminance valuesY_(ref_DS). Similarly, the downscaling function 1004 receives theluminance values Y_(nonref) of the non-reference image and downscalesthe luminance values to produce downscaled luminance valuesY_(nonref_DS). The downscaling allows less data to be processed insubsequent operations, which can help to speed up the subsequentoperations. Any suitable amount of downscaling can be used, such as bydownscaling the data by a factor of four. However, downscaling is notnecessarily required here.

A difference function 1006 identifies the differences between thedownscaled luminance values (or of the original luminance values) on apixel-by-pixel basis. Assuming there is no movement between the twoimages and proper equalization of the images' exposures, the differencefunction 1006 outputs a difference map identifying only the differencesbetween the images, which (ideally) represent motion within the images.For example, the difference map could have darker pixels indicatinglittle difference between the image pixel values and brighter pixelsindicating more differences between the image pixel values. A histogramfunction 1008 generates a histogram based on the difference map, whichquantifies motion statistics within a tile.

A threshold/transfer function 1010 receives the motion statistics fromthe histogram function 1008 and the noise level estimate Sig_Est. Thethreshold/transfer function 1010 uses the noise level estimate toidentify when differences detected in the images are actuallyrepresentative of motion in the images. The output of thethreshold/transfer function 1010 is a motion multiplier 1012.

FIG. 11 illustrates an example implementation of the main block 904 inFIG. 9. As shown in FIG. 11, the main block 904 includes an edgestrength filter 1102 and a main sub-block 1104. The edge strength filter1102 receives the luminance values Y_(ref) of the reference image, thenoise level estimate Sig_Est, and the motion multiplier Mot_Mult andgenerates a norm map, which is used by the main sub-block 1104. Oneexample implementation of the edge strength filter 1102 is describedbelow, although other implementations of the edge strength filter 1102could also be used. The main sub-block 1104 receives the luminance andchrominance values YUV_(ref) and YUV_(nonref) of the reference andnon-reference images, along with the norm map. The main sub-block 1104uses this information to generate the de-ghosting map 906. One exampleimplementation of the main sub-block 1104 is described below, althoughother implementations of the main sub-block 1104 could also be used.

FIG. 12 illustrates an example implementation of the edge strengthfilter 1102 of the main block 904 in FIG. 11. As shown in FIG. 12, theedge strength filter 1102 includes a downscaling function 1202, whichreceives the luminance values Y_(ref) of the reference image anddownscales the luminance values to produce downscaled luminance valuesY_(ref_DS). Any suitable downscaling can be used here (such asdownscaling by a factor of four), although no downscaling may be needed.The downscaled luminance values Y_(ref_DS) are passed through ahigh-pass filter 1204 to produce edge values (denoted Y_(ESF)), whichrepresent rough edges in the scene. The edge values are passed through alow-pass filter 1206 to produce filtered edge values (denotedFilter_(ESF)), which represent smoothed edges in the scene. Thehigh-pass filter 1204 represents any suitable high-pass filter forfiltering pixel values, such as a 3×3 high-pass filter. The low-passfilter 1206 represents any suitable low-pass filter for filtering pixelvalues, such as a 5×5 low-pass filter.

The filtered edge values are provided to an add/shift/multiply function1208, which also receives the noise level estimate Sig_Est and themotion multiplier Mot_Mult. The add/shift/multiply function 1208operates to generate the norm map using this information, where the normmap is used to normalize the motion due to pixel differences within atile as described below. The add/shift/multiply function 1208 can usethe filtered edge values Filter_(ESF), noise level estimate Sig_Est, andmotion multiplier Mot_Mult in any suitable manner to generate the normmap. In some embodiments, the add/shift/multiply function 1208 generatesthe norm map by performing the following calculation, although othersuitable calculations could also occur.((Sig_Est+Filter_(ESF))*Mot_Multi/4)/16  (1)

FIG. 13 illustrates an example implementation of the main sub-block 1104of the main block 904 in FIG. 11. As shown in FIG. 13, the mainsub-block 1104 includes difference functions 1302 and 1304. Thedifference function 1302 identifies the differences Y_(diff) between theluminance values Y_(ref) and Y_(nonref) of the reference andnon-reference images, and the difference function 1304 identifies thedifferences U_(diff) and V_(diff) between the chrominance valuesUV_(ref) and UV_(nonref) of the reference and non-reference images. Thedifferences Y_(diff) in the luminance values are provided to anaverage/downscale function 1306, which averages sets of luminance valuedifferences to downscale the size of the luminance value differences andproduce downscaled luminance value differences Y_(diff_DS). Again, anysuitable downscaling can be used here (such as downscaling by a factorof four), although no downscaling may be needed.

A sum/cap function 1308 receives the downscaled luminance valuedifferences Y_(diff_DS) and the chrominance value differences U_(diff)and V_(diff) and operates to generate the difference map, whichidentifies the differences between the images. The sum/cap function 1308can use the downscaled luminance value differences Y_(diff_DS) andchrominance value differences U_(diff) and V_(diff) in any suitablemanner to generate the difference map. In some embodiments, the sum/capfunction 1308 generates the difference map by performing the followingcalculation, although other suitable calculations could also occur.Diff=(Y _(diff_DS)+(U _(diff) +V _(diff))/2)²  (2)Diff_map=Diff*(Y _(ref)<Sat_Thr)  (3)where Diff_map represents the difference map pixel values and Sat_Thrrepresents a saturation threshold.

The difference map is provided to a low-pass filter (LPF)/dividefunction 1310, which also receives the norm map and two scalar values.One scalar value represents a reference weight Ref_weight, and the otherscalar value represents a weight multiplier W_mult. The low-passfilter/divide function 1310 uses the difference map, norm map, andscalar values to generate the de-ghosting map, which identifies areas inthe images where motion is occurring. The low-pass filter/dividefunction 1310 can use the difference map, norm map, and scalar values inany suitable manner to generate the de-ghosting map. In someembodiments, the low-pass filter/divide function 1310 generates thede-ghosting map by calculating the following, although other suitablecalculations could also occur.Filt_Mot=LPF(Diff_map)/Norm_map  (4)Deghost_map=Ref_weight−min(Ref_weight·Filt_Mot*W_mult)  (5)where Deghost_map represents the de-ghosting map pixel values and LPF( )represents a filtering function. The reference weight Ref_weight heredefines the maximum value that the de-ghosting map pixels can obtain.The weight multiplier W_mult here defines the value that the Filt_Motvalue is multiplied by in order to identify the amount to subtract fromthe reference weight Ref_weight when motion is present. Larger values ofthe weight multiplier W_mult therefore result in larger valuessubtracted from the reference weight Ref_weight, resulting in moremotion being detected.

Note that the process shown in FIGS. 9, 10, 11, 12, and 13 can berepeated for each non-reference image in the collection of alignedimages, typically using the same image from the collection as thereference image. The results from performance of the process in FIGS. 9,10, 11, 12, and 13 is ideally a set of de-ghosting maps 906 thatidentify all of the motion between the non-reference images and thereference image (or at least all motion exceeding a threshold).

FIGS. 14 and 15 illustrate an example process for an image blendingoperation 516 in the process 500 of FIG. 5 in accordance with thisdisclosure. As described above, the image blending operation 516 is usedto blend different portions of images captured by the electronic device101. As shown in FIG. 14, the image blending operation 516 receivesdifferent images 504′, 506′, and 508′ (which represent the alignedversions of the images 504, 506, and 508). The image blending operation516 also receives three blending maps W₁, W₂, and W₃ since there arethree images being used here, although other numbers of blending mapscan be used if two images or more than three images are being blended.As noted above, the blending maps can be based on or can be a compositeof (such as products of) the well-exposedness maps and the de-ghostingmaps generated earlier.

One of the images 504′, 506′, and 508′ in FIG. 14 is treated as thereference image. In this example, the image 506′ is the reference image,which is consistent with earlier uses of the images 506 as the referenceimage. The image 504′ is provided to a multiplier function 1402, whichmultiplies the pixels of the image 504′ by weights in the blending mapW₁. A histogram match function 1404 generates a version of the image506′ using the image 504′ as a reference. Effectively, a transferfunction is applied to the image 506′ in order to make the histogram ofthe image 506′ match the histogram of the image 504′ as closely aspossible. The resulting version of the image 506′ is provided to amultiplier function 1406, which multiplies the pixels of the resultingversion of the image 506′ by weights in a blending map calculated as1−W₁ (assuming each weight in the blending map has a value between zeroand one, inclusive). The results from the multiplier functions 1402 and1406 are summed by an adder function 1408. This alpha-blends the image504′ and a version of the image 506′, synthesizing a new image thathelps to avoid ghosting artifacts by removing motion between the images.Assuming the image 504′ has a shorter exposure than the image 506′, thenew synthesized image may be referred to as a new short-exposure image.

Similarly, the image 508′ is provided to a multiplier function 1410,which multiplies the pixels of the image 508′ by weights in the blendingmap W₃. A histogram match function 1412 generates a version of the image506′ using the image 508′ as a reference. Effectively, a transferfunction is applied to the image 506′ in order to make the histogram ofthe image 506′ match the histogram of the image 508′ as closely aspossible. The resulting version of the image 506′ is provided to amultiplier function 1414, which multiplies the pixels of the resultingversion of the image 506′ by weights in a blending map calculated as1−W₃ (assuming each weight in the blending map has a value between zeroand one, inclusive). This alpha-blends the image 508′ and a version ofthe image 506′, synthesizing a new image that helps to avoid ghostartifacts by removing motion between the images. Assuming the image 508′has a longer exposure than the image 506′, the new synthesized image maybe referred to as a new long-exposure image.

The blended image output from the adder 1408 (such as the newshort-exposure image) is provided to a multiplier function 1418, whichpyramid multiplies the pixels of the new short-exposure image by theweights in the blending map W₁. The image 506′ (as the reference image)is provided to a multiplier function 1420, which pyramid multiplies thepixels of the image 506′ by the weights in the blending map W₂. Theblended image output from the adder 1416 (such as the new long-exposureimage) is provided to a multiplier function 1422, which pyramidmultiplies the pixels of the new long-exposure image by the weights inthe blending map W₃. This weights the three images according to thethree blending maps W₁, W₂, and W₃, respectively. The results arecombined in a pyramid add operation 1424, which combines the results toproduce a final image. Among other things, the multiplier functions1418, 1420, and 1422 and the add operation 1424 operate to pyramid blendthe images to brighten dark regions (such as the background) and recoversaturated regions (such as in the foreground) of the original images504, 506, and 508.

An example of the pyramid blending of a new short-exposure image, animage 506′, and a new long-exposure image is shown in FIG. 15. In thisexample, the image 506′ is referred to as a “medium” image since itsexposure is between the exposures of the images 504′ and 508′. As shownin FIG. 15, the image 506′ is decomposed into a Laplacian pyramid 1502,and the blending map W₂ is decomposed into a Gaussian pyramid 1504. Thedecomposition of an image into a Laplacian pyramid can occur bymultiplying the image data with a set of transform functions. Theblending map W₂ is based on the well-exposedness map and the de-ghostingmap associated with the original image 506. The decomposition of ablending map into a Gaussian pyramid can occur by weighting the blendingmap using Gaussian averages. Each of the pyramids 1502 and 1504 here isa multi-scale pyramid representing an image or blending map at multipleresolution levels or scales. The levels or scales of the pyramids 1502and 1504 are multiplied together as shown in FIG. 15, which representsthe multiplier function 1420. Optionally, at least some of the levels orscales of the pyramids 1502 and 1504 can be multiplied by a halo-controlterm, which is done for halo suppression.

Similar operations occur for the new short- and long-exposure images. Inthis example, the new short-exposure image is decomposed into aLaplacian pyramid 1506, and the blending map W₁ is decomposed into aGaussian pyramid 1508. The blending map W₁ is based on thewell-exposedness map and the de-ghosting map associated with theoriginal image 504. The levels or scales of the pyramids 1506 and 1508are multiplied together (which represents the multiplier function 1418),and optionally at least some of the levels or scales of the pyramids1506 and 1508 can be multiplied by a halo-control term for halosuppression. Also in this example, the new long-exposure image isdecomposed into a Laplacian pyramid 1510, and the blending map W₃ isdecomposed into a Gaussian pyramid 1512. The blending map W₃ is based onthe well-exposedness map and the de-ghosting map associated with theoriginal image 508. The levels or scales of the pyramids 1510 and 1512are multiplied together (which represents the multiplier function 1422),and optionally at least some of the levels or scales of the pyramids1510 and 1512 can be multiplied by a halo-control term for halosuppression.

The resulting products of the pyramids 1502 and 1504, pyramids 1506 and1508, and pyramids 1510 and 1512 are summed at each level or scale(which represents the add operation 1424) to produce a blended imagepyramid 1514. The various levels or scales of the blended image pyramid1514 can then be collapsed or recomposed to produce a blended image of ascene, where the blended image represents a blended version of the newshort-exposure image, the image 506′, and the new long-exposure image.Ideally, the blended image includes or is based on well-exposed portionsof the scene from the image 506′ and from the new short- andlong-exposure images. This may allow, for example, brighter portions ofthe background from the image 504′ to be combined with well-exposedportions of the foreground in the image 506′ in order to produce ablended image with more uniform illumination.

FIG. 16 illustrates an example process for a contrast enhancementoperation 520 in the process of FIG. 5 in accordance with thisdisclosure. As described above, the contrast enhancement operation 520increases the overall contrast of the blended image while maintainingnatural hue within the blended image. As shown in FIG. 16, the contrastenhancement operation 520 includes a histogram equalization function1602, which generally adapts a blended image produced to a suitablescene. In some embodiments, the histogram equalization function 1602uses a global version of contrast-limited adaptive histogramequalization (CLAHE) to improve contrast of the blended image in theluminance domain.

In particular embodiments, histogram equalization is applied on top ofthe tone curve for the blended image, where the parameters to thehistogram equalization function 1602 include a clip limit, a minimumvalue, and a maximum value. The clip limit controls the threshold abovewhich histogram entries are redistributed to other areas of thehistogram. In some cases, the clip limit can have a typical useful rangebetween 0.0 and 0.2. The minimum value represents a contrast controlparameter defining the percentage below which pixels are clipped at avalue of zero. In some cases, the minimum value can have a typicaluseful range between 0.0 and 0.01. The maximum value represents acontrast control parameter defining the percentage above which pixelsare clipped at a value of 255. In some cases, the maximum value can havea typical useful range between 0.99 and 1.0.

The contrast enhancement operation 520 also includes a chroma gainfunction 1604 and a hue correction function 1606. The chroma gainfunction 1604 generally operates to identify the gain applied to theluminance values by the histogram equalization function 1602 and toapply the same gain to the chrominance values of the blended image. Thiscan be done to help avoid color desaturation. However, a visibleartifact can be created when applying a chroma gain globally in theblended image. In particular, there can be a global shift of hue towardsred when applying a chroma gain globally. Hence, the hue correctionfunction 1606 can be applied to correct this global shift. The output ofthe hue correction function 1606 can represent a final image 522 of ascene being captured using the electronic device 101.

Although FIG. 5 illustrates one example of a process 500 for multi-scaleblending of images in a mobile electronic device and FIGS. 6 through 16illustrate examples of operations in the process 500 of FIG. 5, variouschanges may be made to FIGS. 5 through 16. For example, while shown assequences of steps, various operations shown in FIGS. 5 through 16 couldoverlap, occur in parallel, occur in a different order, or occur anynumber of times. Also, the specific operations shown in FIGS. 6 through16 are examples only, and other techniques could be used to perform eachof the operations shown in FIGS. 6 through 16.

It should be noted that the operations shown in FIGS. 2 through 16 canbe implemented in an electronic device 101 in any suitable manner. Forexample, in some embodiments, the operations shown in FIGS. 2 through 16can be implemented or supported using one or more software applicationsor other software instructions that are executed by the processor 120 ofthe electronic device 101. In other embodiments, at least some of theoperations shown in FIGS. 2 through 16 can be implemented or supportedusing dedicated hardware components. In general, the operations shown inFIGS. 2 through 16 can be performed using any suitable hardware or anysuitable combination of hardware and software/firmware instructions.

It should also be noted that the operations shown in FIGS. 2 through 16are described above as being performed using a specific implementationof the electronic device 101, but a number of modifications can be madebased on the actual electronic device 101 being used. For example, theelectronic device 101 could include or use a single camera or multiplecameras. If a single camera is used, multiple images of a scene could becaptured sequentially, such as in one or more fast bursts. If multiplecameras are used, it may be possible to capture multiple imagesconcurrently or in an overlapping manner, such as by capturing multipleimages of a scene at the same time but with different camera exposuresusing different cameras. If needed, multiple images of the scene couldstill be captured sequentially using at least one of the multiplecameras. Assuming the geometry of the multiple cameras is known ahead oftime, this geometry can be used to help align the images captured by thecameras or perform other functions. As another example, the electronicdevice 101 is described above as performing various operations using YUVimage data. However, data in other domains (such as RGB data) could alsobe used or processed. As a third example, the techniques described inthis patent document could be combined with conventional HDR imageprocessing algorithms, such as in a software library used by theelectronic device 101. This may allow a user of the electronic device101 to select between different image processing algorithms, such asbased on the specific situation or based on user preference.

FIG. 17 illustrates an example method 1700 for multi-pair image analysisand multi-scale blending in accordance with this disclosure. For ease ofexplanation, the method 1700 shown in FIG. 17 is described as beingperformed using the electronic device 101 of FIG. 1 and the techniquesshown in FIGS. 2 through 16. However, the method 1700 shown in FIG. 17could be used with any other suitable electronic device and in anysuitable system, and various steps in the method 1700 may or may notoccur using the operations and functions shown in FIGS. 2 through 16.

As shown in FIG. 17, multiple ambient images of a scene are capturedusing an electronic device and without using a flash at step 1702. Thiscould include, for example, the processor 120 of the electronic device101 receiving a capture request 202 and causing at least one camera ofthe electronic device 101 to capture the ambient images 206 of thescene. This could also include the processor 120 of the electronicdevice 101 controlling the camera(s) to use different exposures whencapturing the ambient images 206. Multiple flash images of the scene arecaptured using the electronic device and while using a pilot flashsequence at step 1704. This could include, for example, the processor120 of the electronic device 101 causing the at least one camera of theelectronic device 101 to capture the flash images 210 of the scene. Thiscould also include the processor 120 of the electronic device 101controlling the flash 190 to produce the pilot flash sequence (possiblyat a predefined flash strength) and controlling the camera(s) to usedifferent exposures when capturing the flash images 210.

Multiple pairs of the captured images are analyzed to estimate theexposure differences obtained using the flash at step 1706. This couldinclude, for example, the processor 120 of the electronic device 101processing multiple pairs of images (each pair including one of theambient images 206 and one of the flashing images 210 having a commonexposure time) to identify the exposure differences between each pair ofimages. Different pairs of images can be captured using different cameraexposures. As a specific example, each pair of images could be processedby dividing the pixel values in the images, converting the quotientsinto a logarithmic domain, applying a rectifier linear unit operation,averaging the resulting values, and performing an edge-preservingfiltering of the averaged values. As another specific example, each pairof images could be processed using an artificial intelligence function(such as a convolutional neural network 406). An appropriate flashstrength for the scene is identified using the exposure differences atstep 1708. This could include, for example, the processor 120 of theelectronic device 101 mapping the identified exposure differences to theappropriate flash strength. As noted above, the mapping can be based ona number of factors.

The flash of the electronic device is fired at the determined flashstrength and additional images of the scene are captured using theelectronic device at step 1710. This could include, for example, theprocessor 120 of the electronic device 101 controlling the flash 190 tofire at the appropriate flash strength determined earlier. This couldalso include the processor 120 of the electronic device 101 causing theat least one camera of the electronic device 101 to capture theadditional images 504, 506, and 508 of the scene. The additional images504, 506, and 508 can be captured using a different exposure but thesame common flash strength. The additional images are aligned andpre-processed at step 1712. This could include, for example, theprocessor 120 of the electronic device 101 aligning the additionalimages 504, 506, and 508 using feature point detection and matching andblock searching. This could also include the processor 120 of theelectronic device 101 performing exposure analysis and de-ghosting ofthe aligned images 504′, 506′, and 508′.

The aligned and pre-processed images are then blended. In this example,the blending occurs by generating multi-scale representations of imagesafter alignment and processing at step 1714, and the multi-scalerepresentations are blended to produce a blended image of the scene atstep 1716. This could include, for example, the processor 120 of theelectronic device 101 decomposing one of the images selected as areference (such as the image 506″) into a Laplacian pyramid anddecomposing the associated blending map into a Gaussian pyramid. Thiscould also include the processor 120 of the electronic device 101generating one or more synthesized images based on one or more processedimages (such as new versions of the image 506″ based on the images 504″and 508″), decomposing the synthesized image(s) into one or moreLaplacian pyramids, and decomposing the associated blending map(s) intoone or more Gaussian pyramids. This can further include the processor120 of the electronic device 101 multiplying each Laplacian pyramid bythe associated Gaussian pyramid, applying any desired halo correctionfactors, and summing the results at each level of the multi-scalerepresentations. In addition, this can include the processor 120 of theelectronic device 101 collapsing to summed results to produce a blendedimage of the scene. Ideally, the blended image of the scene has a moreuniform illumination compared to any of the original images.

Any desired post-processing of the blended image occurs at step 1718.This could include, for example, the processor 120 of the electronicdevice 101 performing an edge-enhanced noise filtering function 518and/or a contrast enhancement operation 520 on the blended image of thescene. The output of the post-processing is a final image of the scene,which can be stored, output, or used in some manner at step 1720. Thiscould include, for example, the processor 120 of the electronic device101 displaying the final image of the scene on the display 160 of theelectronic device 101. This could also include the processor 120 of theelectronic device 101 saving the final image of the scene to a cameraroll stored in a memory 130 of the electronic device 101. This couldfurther include the processor 120 of the electronic device 101 attachingthe final image of the scene to a text message, email, or othercommunication to be transmitted from the electronic device 101. Ofcourse, the final image of the scene could be used in any other oradditional manner.

Although FIG. 17 illustrates one example of a method 1700 for multi-pairimage analysis and multi-scale blending, various changes may be made toFIG. 17. For example, while shown as a series of steps, various steps inFIG. 17 could overlap, occur in parallel, occur in a different order, oroccur any number of times. Also, it is not necessary that the techniquesfor multi-pair image analysis described above be used with thetechniques for multi-scale blending described above. It is possible, forinstance, to perform the multi-pair image analysis to identify and firethe flash 190 without then performing the multi-scale blending.Similarly, it is possible to perform the multi-scale blending based onimages captured using a flash strength that is not based on themulti-pair image analysis.

FIGS. 18, 19, 20, and 21 illustrate example results that can be obtainedusing multi-pair image analysis and multi-scale blending in accordancewith this disclosure. In FIG. 18, an image 1800 of a scene that includesa darker foreground with an object (a teddy bear resting on a table) anda brighter background is captured, where no flash is used. As can beseen here, the brighter background is generally well-illuminated, whilethe foreground with the object is dark and under-exposed. In FIG. 19, animage 1900 of the same scene is captured, but this time a standard flashis used. As can be seen here, the brighter background is now darker,while the foreground with the object is very bright and likelyconsidered over-exposed. In FIG. 20, an image 2000 of the same scene iscaptured, but this time a standard flash and standard HDR imageprocessing are used. As can be seen here, the background is stillbright, and slight improvements have been made in illuminating theforeground. Unfortunately, the foreground still has significantillumination differences with the background.

In FIG. 21, an image 2100 of the same scene is captured using theapproaches described above for multi-pair image analysis and multi-scaleblending. As can be seen here, the background of the scene is stillwell-illuminated, but the background has not been darkened as was donein the standard flash image 1900 in FIG. 19. Moreover, the foreground ofthe scene has been brightened compared to the original non-flash image1800 in FIG. 18, but the foreground is not over-exposed as in thestandard flash image 1900 in FIG. 19. Also, the foreground of the scenehas a more consistent illumination compared to the background, which isnot achieved using the standard HDR image processing for the image 2000in FIG. 20. The image 2100 is therefore more aesthetically pleasingoverall compared to any of the images 1800, 1900, and 2000.

Although FIGS. 18, 19, 20, and 21 illustrate examples of results thatcan be obtained using multi-pair image analysis and multi-scaleblending, various changes may be made to FIGS. 18, 19, 20, and 21. Forexample, FIGS. 18, 19, 20, and 21 are merely meant to illustrate oneexample of the type of results that could be obtained using theapproaches described in this disclosure. Obviously, standard, flash, andHDR images can vary widely, and the results obtained using theapproaches described in this patent document can also vary widelydepending on the circumstances.

Although this disclosure has been described with reference to variousexample embodiments, various changes and modifications may be suggestedto one skilled in the art. It is intended that this disclosure encompasssuch changes and modifications as fall within the scope of the appendedclaims.

What is claimed is:
 1. A method comprising: capturing multiple ambientimages of a scene using at least one camera of an electronic device andwithout using a flash of the electronic device; capturing multiple flashimages of the scene using the at least one camera of the electronicdevice and during firing of a pilot flash sequence using the flash;analyzing multiple pairs of images to estimate exposure differencesobtained using the flash, wherein each pair of images includes one ofthe ambient images and one of the flash images that are both capturedusing a common camera exposure, and wherein different pairs of imagesare captured using different camera exposures; determining a flashstrength for the scene based on the estimate of the exposuredifferences; and firing the flash based on the determined flashstrength.
 2. The method of claim 1, wherein analyzing each pair ofimages comprises at least one of: identifying exposure differencesbetween the ambient image and the flash image in the pair at a pixellevel; identifying exposure differences between the ambient image andthe flash image in the pair for a foreground and a background of theimages; and identifying exposure differences between the ambient imageand the flash image in the pair for one or more objects in the images.3. The method of claim 1, wherein determining the flash strength for thescene comprises mapping the identified exposure differences to the flashstrength.
 4. The method of claim 1, further comprising: capturingadditional images of the scene using the at least one camera of theelectronic device, at least some of the additional images captured atthe determined flash strength and using different camera exposures; andblending the additional images to produce a blended image of the scene.5. The method of claim 4, wherein blending the additional imagescomprises: aligning the additional images; generating exposure maps forthe aligned additional images using different metrics for the differentcamera exposures of the additional images; and generating motion mapsfor the aligned additional images or for equalized versions of thealigned additional images; and wherein the blended image is producedusing the exposure maps, synthesized images based on the aligned images,and the motion maps.
 6. The method of claim 4, wherein: the additionalimages include first, second, and third additional images; and blendingthe additional images comprises: blending a version of the firstadditional image and a first histogram-matched version of the secondadditional image to produce a fourth additional image; blending aversion of the third additional image and a second histogram-matchedversion of the second additional image to produce a fifth additionalimage; and blending the second, fourth, and fifth additional images toproduce the blended image.
 7. The method of claim 4, wherein: theblended image includes at least one object in the scene or a foregroundof the scene illuminated using the determined flash strength; and the atleast one object in the scene or the foreground of the scene and abackground of the scene have a more uniform illumination than in theadditional images of the scene.
 8. An electronic device comprising: atleast one camera; a flash; and at least one processing device configuredto: capture multiple ambient images of a scene using the at least onecamera and without using the flash; capture multiple flash images of thescene using the at least one camera and during firing of a pilot flashsequence using the flash; analyze multiple pairs of images to estimateexposure differences obtained using the flash, wherein each pair ofimages includes one of the ambient images and one of the flash imagesthat are both captured using a common camera exposure, and whereindifferent pairs of images are captured using different camera exposures;determine a flash strength for the scene based on the estimate of theexposure differences; and fire the flash based on the determined flashstrength.
 9. The electronic device of claim 8, wherein, to analyze eachpair of images, the at least one processing device is configured to atleast one of: identify exposure differences between the ambient imageand the flash image in the pair at a pixel level; identify exposuredifferences between the ambient image and the flash image in the pairfor a foreground and a background of the images; and identify exposuredifferences between the ambient image and the flash image in the pairfor one or more objects in the images.
 10. The electronic device ofclaim 8, wherein, to determine the flash strength for the scene, the atleast one processing device is configured to map the identified exposuredifferences to the flash strength.
 11. The electronic device of claim 8,wherein the at least one processing device is further configured to:capture additional images of the scene using the at least one camera, atleast some of the additional images captured at the determined flashstrength and using different camera exposures; and blend the additionalimages to produce a blended image of the scene.
 12. The electronicdevice of claim 11, wherein, to blend the additional images, the atleast one processing device is configured to: align the additionalimages; generate exposure maps for the aligned additional images usingdifferent metrics for the different camera exposures of the additionalimages; and generate motion maps for the aligned additional images orfor equalized versions of the aligned additional images; and wherein, toproduce the blended image, the at least one processing device isconfigured to use the exposure maps, synthesized images based on thealigned images, and the motion maps.
 13. The electronic device of claim11, wherein: the additional images include first, second, and thirdadditional images; and to blend the additional images, the at least oneprocessing device is configured to: blend a version of the firstadditional image and a first histogram-matched version of the secondadditional image to produce a fourth additional image; blend a versionof the third additional image and a second histogram-matched version ofthe second additional image to produce a fifth additional image; andblend the second, fourth, and fifth additional images to produce theblended image.
 14. The electronic device of claim 11, wherein: theblended image includes at least one object in the scene or a foregroundof the scene illuminated using the determined flash strength; and the atleast one object in the scene or the foreground of the scene and abackground of the scene have a more uniform illumination than in theadditional images of the scene.
 15. A non-transitory machine-readablemedium containing instructions that when executed cause at least oneprocessor of an electronic device to: capture multiple ambient images ofa scene using at least one camera of the electronic device and withoutusing a flash of the electronic device; capture multiple flash images ofthe scene using the at least one camera of the electronic device andduring firing of a pilot flash sequence using the flash; analyzemultiple pairs of images to estimate exposure differences obtained usingthe flash, wherein each pair of images includes one of the ambientimages and one of the flash images that are both captured using a commoncamera exposure, and wherein different pairs of images are capturedusing different camera exposures; determine a flash strength for thescene based on the estimate of the exposure differences; and fire theflash based on the determined flash strength.
 16. The non-transitorymachine-readable medium of claim 15, wherein the instructions that whenexecuted cause the processor to analyze each pair of images compriseinstructions that when executed cause the processor to at least one of:identify exposure differences between the ambient image and the flashimage in the pair at a pixel level; identify exposure differencesbetween the ambient image and the flash image in the pair for aforeground and a background of the images; and identify exposuredifferences between the ambient image and the flash image in the pairfor one or more objects in the images.
 17. The non-transitorymachine-readable medium of claim 15, wherein the instructions whenexecuted further cause the processor to: capture additional images ofthe scene using the at least one camera, at least some of the additionalimages captured at the determined flash strength and using differentcamera exposures; and blend the additional images to produce a blendedimage of the scene.
 18. The non-transitory machine-readable medium ofclaim 17, wherein the instructions that when executed cause theprocessor to blend the additional images comprise instructions that whenexecuted cause the processor to: align the additional images; generateexposure maps for the aligned additional images using different metricsfor the different camera exposures of the additional images; andgenerate motion maps for the aligned additional images or for equalizedversions of the aligned additional images; and wherein the instructionsthat when executed cause the processor to produce the blended imagecomprise instructions that when executed cause the processor to use theexposure maps, synthesized images based on the aligned images, and themotion maps.
 19. The non-transitory machine-readable medium of claim 17,wherein: the additional images include first, second, and thirdadditional images; and the instructions that when executed cause theprocessor to blend the additional images comprise instructions that whenexecuted cause the processor to: blend a version of the first additionalimage and a first histogram-matched version of the second additionalimage to produce a fourth additional image; blend a version of the thirdadditional image and a second histogram-matched version of the secondadditional image to produce a fifth additional image; and blend thesecond, fourth, and fifth additional images to produce the blendedimage.
 20. The non-transitory machine-readable medium of claim 17,wherein: the blended image includes at least one object in the scene ora foreground of the scene illuminated using the determined flashstrength; and the at least one object in the scene or the foreground ofthe scene and a background of the scene have a more uniform illuminationthan in the additional images of the scene.